Mobile computing devices including smart phones and tablet computers serve many functions in business, government, and personal computing. The essential aspects of mobile computing devices as we understand them today include a highly portable form factor enabling ease of use in many settings, usually with a touch screen interface and often combining telephone features; a combination of sensors and networking to enable many useful functions; and the ability to use dedicated “apps” (i.e., computer application software programs designed for mobile computing devices) that leverage the sensors and user interface to perform specific useful functions for the device user.
For example, an app may be provided that executes on mobile device operating systems such as Apple Inc.'s iOS®, Google Inc.'s Android®, or Microsoft Inc.'s Windows 10 Mobile®. These platforms typically provide frameworks that allow apps to communicate with one another and with particular hardware and software components of mobile computing devices. For example, the mobile operating systems named above each provide frameworks enabling apps to interact with location services circuitry, wired and wireless network interfaces, user contacts, and other services. Communication with hardware and software modules executing outside of the app is typically provided via application programming interfaces (APIs) provided by the mobile computing device operating system.
One consequence of the proliferation and increased capabilities of mobile computing devices is that it has become possible for apps to gather and transmit extensive data, on various networks, regarding the user of the mobile computing devices. This data can include very sensitive information, such as the user's location, photos, communications, personal identification numbers, passwords, and both financial and health information. The nature of the information accessible by mobile computing devices illustrates the growing need to enhance the security of the various apps installed on a given mobile computing device, which requires mobile app security testing.
The inventors have discovered latent and unmet needs stemming from the nature of existing mobile app security testing methods. User interface (UI) testing may be performed manually by a human tester, or it can be performed automatically with the use of software. Either way, it is typical to utilize a testing device (or “rig”) that connects to a mobile computing device that hosts the mobile app. Testing may involve recording, by the rig, a user's interaction with the mobile app. Historically, this involves pixel-based interactions, because the tester interacts on the rig with a ported display of the UI of the mobile app. For example, a user pressing a submit button on a login screen of the mobile app may be recorded based on x, y pixel coordinates, and the record of the interaction may be something like “press at pixel 200, 1400 and release at pixel 360, 1400.” But because pixel-based testing focuses on the exact location of an object on the screen, if an object's appearance or location within a display changes based on screen resolution, display orientation, or the like, some tests may fail. Accordingly, although manual testing by a human is time consuming, the human user has historically been better able to navigate a UI, because of course, mobile apps are typically designed for human interaction, and humans can more effectively adjust to changes in resolution, display orientation, or the like caused by porting the display of a mobile app UI from a mobile computing device to a rig. However, manual testing can introduce inconsistency, delays, and errors into the mobile app security testing process.